Resolved: Logs csv to JSON or any mechanism to get metrics

In this post, we will see how to resolve Logs csv to JSON or any mechanism to get metrics

Question:

i have a log file and i need to extract stats based on logs captured.
Sample log format:

2023-03-09 18:55:56,INFO,capturing: DataIngestion
2023-03-09 18:55:57,INFO,waiting to get data
2023-03-09 18:56:58,INFO,time started,2023-03-09 18:56:57
2023-03-09 18:56:59,INFO,data received processing started
2023-03-09 18:56:59,INFO,data to convert, 23000
2023-03-09 18:56:00,INFO,covert json,2023-03-09 18:56:57
2023-03-09 18:57:00,INFO,convesion time,0 days 00:01:00
2023-03-09 18:57:58,INFO,process data,2023-03-09 18:57:58
2023-03-09 18:59:01,INFO,process completed,0 days 00:02:03
2023-03-09 18:59:10,INFO,time taken,0 days 00:00:09
2023-03-09 18:59:02,INFO,waiting to get data
2023-03-09 18:59:03,INFO,time started,2023-03-09 18:59:03
2023-03-09 18:59:59,INFO,data received processing started
2023-03-09 18:59:59,INFO,data to convert,30000
2023-03-09 19:00:01,INFO,covert json,2023-03-09 19:00:01
2023-03-09 19:01:31,INFO,convesion time,0 days 00:01:30
2023-03-09 19:01:32,INFO,process data,2023-03-09 19:01:32
2023-03-09 19:04:30,INFO,process completed,0 days 00:03:28
2023-03-09 19:04:31,INFO,time taken,0 days 00:03:31

One iteration should be from “data received processing started” to “time taken”
I need to get

Total data, conversion time and process time to complete and total time taken.
Min data, Max data, Avg data — respective conversion time taken for min data, max data and Avg data(for suppose there are 10 iteration – in that min data in that 10 iterations of data, max data in that 10 iterations of data and Average of 10 iterations and their respective conversion, process and time taken)

I started converting csv to json first and thought of extracting from there but am stuck to move further. here is what i tired,

import csv, json

csvfile = open('test1.log', 'r')
jsonfile = open('file.json', 'w')


fieldnames = ("logtime","log","type","time stats")
reader = csv.DictReader( csvfile, fieldnames)
for row in reader:
    json.dump(row, jsonfile)
    jsonfile.write('\n')

Please suggest me how to move further to get stats or is there any better way to get stats from my provided logs

Best Answer:

You can use pandas to read your csv and pivot the segments corresponding to entries then concat:

import pandas as pd

df = pd.read_csv('log.csv', names=['time', 'type', 'event', 'value']).drop(columns='time')

entry_start = df[df['event']=="data received processing started"].index
entry_end = df[df['event']=="time taken"].index + 1

df = pd.concat([
        df[s:e].pivot(index='type', columns='event', values='value')
        for s, e in zip(entry_start, entry_end)]
    ).drop(columns="data received processing started")

This will give you a dataframe of the following form:

event   convesion time          covert json data to convert process completed         process data       time taken
type                                                                                                               
INFO   0 days 00:01:00  2023-03-09 18:56:57           23000   0 days 00:02:03  2023-03-09 18:57:58  0 days 00:00:09
INFO   0 days 00:01:30  2023-03-09 19:00:01           30000   0 days 00:03:28  2023-03-09 19:01:32  0 days 00:03:31

You can then easily access min, max and avg:

# make sure the column is of type int
df['data to convert'] = df['data to convert'].astype(int)
data_min = df['data to convert'].min()
data_max = df['data to convert'].max()
data_avg = df['data to convert'].mean()

And the corresponding conversion time:

df.loc[df['data to convert']==data_min, 'convesion time']

Edit: adding time taken to the output, you can sum all column elements:

# converting column to timedelta
df['time taken'] = pd.to_timedelta(df['time taken'])
df['time taken'].sum()

If you have better answer, please add a comment about this, thank you!

Source: Stackoverflow.com

Resolved: Logs csv to JSON or any mechanism to get metrics

Resolved: Is pandas groupby() function always produce a DataFrame with the same order respect to the column we group by?

Resolved: Kivy widget hierarchy not behaving as expected

Resolved: Pandas Groupby Get Values from Previous Group

Question:

Best Answer:

Related Posts

Resolved: Is pandas groupby() function always produce a DataFrame with the same order respect to the column we group by?

Resolved: Kivy widget hierarchy not behaving as expected

Resolved: Pandas Groupby Get Values from Previous Group