Skip to content

Parses JSON log files from S3 and stores them in an SQL database of adaptive (i.e. auto-expanding) schema

Notifications You must be signed in to change notification settings

Brickflow/s3json2sql

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 

Repository files navigation

s3json2sql

This library fetches files from an s3 bucket and parses them as JSON log entries (where each row is a JSON object to parse).

Usage

    var s3json2sql = require('s3json2sql');
    s3json2sql({
      s3: {
        s3Options: {
          accessKeyId: '...',
          secretAccessKey: '...',
          ...
        },
        bucket: 'brickflow-logs'
      },
      sql: {
        uri: 'mysql://root:[email protected]/logs'
      },
      tmpDir: __dirname + '/tmp'
    });

Note that this utility is aimed to be run by forever or similar.

About

Parses JSON log files from S3 and stores them in an SQL database of adaptive (i.e. auto-expanding) schema

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published