Seq Documentation and Support

Welcome to the Seq documentation hub. You'll find comprehensive guides and documentation to help you start working with Seq as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    
Ask A Question

Questions

17

Query Syntax - Using Aggregate Function 'Distinct' performance on large datasets?

Hello Support, In my NET Core project I am building a log report template UI so users can query logs from within my own WebApp. One of the nifty concepts i came up with in my previous project (before i started using Seq) was searching the MS Sql database where the logs were stored and for each log event property, I would scan the whole table column in SQL using a distinct query in order to return the results of the actual values. These values I would then list within a drop down list in the web form, so that the user could select their own choice of event property values to use as part of their search when querying the logs database. The beauty of doing this is the logs event property values can change/evolve over time without having to hard code anything in the software i.e If I start writing new property values in my C# code, these will be pulled through to the dropdown list. I've looked at how to do this using an Seq query through your SDK, and noticed a couple of things I would appreciate some input on: Q. It looks like we have to set a start DateTime for the Seq query to work. In this scenario, I would simply set a start date back far enough that we always know we're querying the dataset from the very beginning of time. It seems to work doing this with a fairly small dataset, is this going to viable as the logs start growing with big data stored? As the logs database starts to fill up, in an enterprise environment, we could imagine that potentially millions of log events might be stored in Seq. Q. What impact on query time/performance would we likely have in performing a 'Distinct' query on a property column in Seq DB? For this to continue being a viable solution, we would need a fairly quick turnaround so the user is not waiting forever before they can select the dropdown lists that are populated with distinct property values. Thanks in advance.

Posted by OLIVER BENTLEY 3 months ago

4

Paging backwards using lastReadEventId is not working

Hello, I'm using your SDK to fetch events from Seq into our own WebApp front end (net core 3.1 Razor) I'm able to page in one direction using a 'Next' button by using the lastReadEventId variable which is returned from the API after each iteration. The result of this is I'm able to click the 'Next' button to produce a list of results that are older in time by timestamp given we display the most recent event first by default (note i'm using the count var as well to do this so I return a set of say 100 rows at a time and show them in a razor page list view) Each time a new unique lastReadEventId value is returned, i'm storing it in a javascript array so i keep track of the page iterations by direction in which way we iterate through the array (forwards or backwards) If the user wants to page backwards i.e. return results that were newer than the previous request, then using javascript logic we pass a previously stored lastReadEventId variable to the API in an attempt to fetch a previous set of results. But this is not returning the newer results as expected. From VS 2019 c# debugging I can see we are passing the correct lastReadEventId in order to return the events from a previous round, but the server does not return correct set. It seems that this lastReadEventId variable is being cached somewhere in Seq and decides for itself what the value should be, as opposed to a more logical cause & affect in simply giving us the results we're asking for. It seems we cannot this var to re-produce a set of results that are more recent in time than the last iteration. Paging only in one direction is not going to give us a workable solution. Could you please advise how we get around this? thanks.

Posted by OLIVER BENTLEY 3 months ago

1

Have to docker-compose down and up about 10 times to get ui to show after system restart

docker-compose.yml ```yml version: "2" services: seq: image: 'datalust/seq:5.1.3004' restart: always container_name: seq_log_server environment: ACCEPT_EULA: Y BASE_URI: ${BASE_URI}:${HOST_HTTP_PORT} SEQ_API_CANONICALURI: ${SEQ_API_CANONICALURI}:${HOST_HTTP_PORT} volumes: - $HOST_PATH_TO_SEQ:/data - ./settings.seqbac:/settings.seqbac ports: - $HOST_HTTP_PORT:80 - $HOST_INGESTION_PORT:5341 ``` Example of one that doesn't show ui. ```bash -bash-4.2$ docker-compose up Creating network "seq_default" with the default driver Creating seq_log_server Attaching to seq_log_server seq_log_server | ──────────────────────────────────────── seq_log_server | Seq ♦ Machine data, for humans. seq_log_server | ─────────── © 2019 Datalust Pty Ltd ──── seq_log_server | seq_log_server | Running as server; press Ctrl+C to exit. seq_log_server | seq_log_server | [23:22:31 INF] Seq "5.1.3004" running on OS "Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019" seq_log_server | [23:22:32 INF] Seq listening on ["http://localhost/", "http://localhost:5341/"] seq_log_server | [23:22:32 INF] Seq using canonical URI "REDACTED" seq_log_server | [23:22:32 INF] Opening document store "/data/Documents/documents.lmdb" seq_log_server | [23:22:32 INF] Opening event store at "/data/Extents" seq_log_server | [23:22:32 INF] Available storage engines in order of preference are ["FLARE"] seq_log_server | [23:22:32 INF] Cache warm-up is required seq_log_server | [23:22:32 INF] Storage subsystem available seq_log_server | [23:22:32 INF] Started "Seq.Apps.GenericHost" process 150 for "appinstance-139" seq_log_server | [23:22:32 INF] Started "Seq.Apps.GenericHost" process 156 for "appinstance-315" seq_log_server | [23:22:32 INF] Started "Seq.Apps.GenericHost" process 161 for "appinstance-346" seq_log_server | [23:22:32 INF] Started "Seq.Apps.GenericHost" process 163 for "appinstance-364" seq_log_server | [23:22:32 INF] Started "Seq.Apps.GenericHost" process 166 for "appinstance-381" seq_log_server | [23:22:32 INF] Started "Seq.Apps.GenericHost" process 168 for "appinstance-394" seq_log_server | [23:22:40 INF] Deduplication buffer renewed on reaching {"Schemata": 99, "Strings": 103401} seq_log_server | [23:22:49 INF] Deduplication buffer renewed on reaching {"Schemata": 184, "Strings": 104604} seq_log_server | [23:22:59 INF] Deduplication buffer renewed on reaching {"Schemata": 193, "Strings": 109754} seq_log_server | [23:23:07 INF] Deduplication buffer renewed on reaching {"Schemata": 197, "Strings": 104219} seq_log_server | [23:23:18 INF] Deduplication buffer renewed on reaching {"Schemata": 194, "Strings": 107967} ``` Example of one that works ```bash -bash-4.2$ docker-compose logs -f Attaching to seq_log_server seq_log_server | ──────────────────────────────────────── seq_log_server | Seq ♦ Machine data, for humans. seq_log_server | ─────────── © 2019 Datalust Pty Ltd ──── seq_log_server | seq_log_server | Running as server; press Ctrl+C to exit. seq_log_server | seq_log_server | [23:38:52 INF] Seq "5.1.3004" running on OS "Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019" seq_log_server | [23:38:53 INF] Seq listening on ["http://localhost/", "http://localhost:5341/"] seq_log_server | [23:38:53 INF] Seq using canonical URI "REDACTED" seq_log_server | [23:38:53 INF] Opening document store "/data/Documents/documents.lmdb" seq_log_server | [23:38:53 INF] Opening event store at "/data/Extents" seq_log_server | [23:38:53 INF] Available storage engines in order of preference are ["FLARE"] seq_log_server | [23:38:53 INF] Cache warm-up is required seq_log_server | [23:38:53 INF] Storage subsystem available seq_log_server | [23:38:53 INF] Started "Seq.Apps.GenericHost" process 141 for "appinstance-139" seq_log_server | [23:38:53 INF] Started "Seq.Apps.GenericHost" process 146 for "appinstance-315" seq_log_server | [23:38:53 INF] Started "Seq.Apps.GenericHost" process 148 for "appinstance-346" seq_log_server | [23:38:53 INF] Started "Seq.Apps.GenericHost" process 158 for "appinstance-364" seq_log_server | [23:38:53 INF] Started "Seq.Apps.GenericHost" process 161 for "appinstance-381" seq_log_server | [23:38:53 INF] Started "Seq.Apps.GenericHost" process 164 for "appinstance-394" seq_log_server | [23:39:02 INF] Deduplication buffer renewed on reaching {"Schemata": 99, "Strings": 103401} seq_log_server | [23:39:12 INF] Deduplication buffer renewed on reaching {"Schemata": 184, "Strings": 104602} seq_log_server | [23:39:21 INF] Deduplication buffer renewed on reaching {"Schemata": 193, "Strings": 109751} seq_log_server | [23:39:29 INF] Deduplication buffer renewed on reaching {"Schemata": 197, "Strings": 104220} ``` NOTE: This happened after I had to move the docker-compose.yml file from one directory to another. Before I did this I stopped and removed the old container. I left the volume because I wanted to keep my logs.

Posted by Shane Swayney 3 months ago