Code

scraper-etl#71

Code

Merged

rafgpereira merged 50 commits intomainunb-mds/2024-1-EducaMinas-backend:mainfrom developunb-mds/2024-1-EducaMinas-backend:developCopy head branch name to clipboardSep 4, 2024

+5,835-2

Commits

Commits on May 3, 2024

• feat(#1): export of csv data filters with combined filters

  johan-rocha

  committed
  f1f0430

Commits on May 7, 2024

• feat(#1): selecting elements for filter combination

  johan-rocha

  committed
  043bef0

Commits on Jul 8, 2024

• fix(#1): rename

  johan-rocha

  committed
  80a71fa

• feat(#1): read csv data

  johan-rocha

  committed
  8bdc78e

• fix(#1): prototypes

  johan-rocha

  committed
  c388f74

Commits on Jul 30, 2024

• feat(#1): renaming csv files by filters and distributing them into folders by category

  johan-rocha

  committed
  93e897c

• feat(#1): extracting municipalities from Excel and inserting them into the database

  johan-rocha

  committed
  74f932a

• feat(#1): WebScrapper conda environment requirements

  johan-rocha

  committed
  7694ea2

• feat(#1): filter reading class

  johan-rocha

  committed
  1eae1e1

Commits on Aug 5, 2024

• feat(#1): working send data for db

  johan-rocha

  committed
  a3b2f07

Commits on Aug 24, 2024

• feat: municipio_id compatibility dictionary

  johan-rocha

  committed
  3b66555

• feat(#42, #43): checkpoint and performance in bulk insertion

  johan-rocha

  committed
  09e3de2

• feat(#1, #2): total values combination sum and insertion in database

  johan-rocha

  committed
  0529f2a

• fix: testing

  johan-rocha

  committed
  a8f9481

• fix: removing commited trash items

  johan-rocha

  committed
  2136b85

• fix: removing trash

  johan-rocha

  committed
  4d2c1b7

• fix: gitignore add items

  johan-rocha

  committed
  f7016b1

• fix: removing test files

  johan-rocha

  committed
  abfee84

Commits on Aug 28, 2024

• .gitignore

  johan-rocha

  committed
  0cbf22a

• fix(#1): rename

  johan-rocha

  committed
  5d51d83

• fix(#1): rename inpynb

  johan-rocha

  committed
  a740898

• fix(#1):change dir

  johan-rocha

  committed
  1378acc

• fix(#1): rename

  johan-rocha

  committed
  979d204

• feat(#67): etl pipeline

  johan-rocha

  committed
  2e0274f

• feat(#67) extractor interface and classes

  johan-rocha

  committed
  9545e64

• feat(#67) transform interface and classes

  johan-rocha

  committed
  425ab8b

• feat(#67) connect interface

  johan-rocha

  committed
  a1e793e

• feat(#67) load interface and classes

  johan-rocha

  committed
  c065eda

• feat(#67) config class

  johan-rocha

  committed
  83a05e2

• feat(#67) orm metadata

  johan-rocha

  committed
  7615929

• feat(#67): __init__.py

  johan-rocha

  committed
  ef5c273

• feat(#67): extract and create JSON Municipios

  johan-rocha

  committed
  f04ee2f

• feat(#67): loading municipio table to DB

  johan-rocha

  committed
  77e15fd

• feat(#43, #44, #67): ETL indicadores

  johan-rocha

  committed
  6875517

• feat(#43, #44, #67): ETL matricula

  johan-rocha

  committed
  a761609

• feat(#67): main.py

  johan-rocha

  committed
  b40237a

• delet files

  johan-rocha

  committed
  44cdbaa

Commits on Sep 4, 2024

• ignore pycache from models

  johan-rocha

  committed
  631f25b

• feat(#67): ETL indicator module

  johan-rocha

  committed
  0b9b227

• fix(#67): connect module new imports from models

  johan-rocha

  committed
  24d7bb6

• feat(#1, #67): refactoring orm

  johan-rocha

  committed
  99342aa

• fix(#1, #67): remove old orm map

  johan-rocha

  committed
  ca7b116

• fix: dotenv abs path

  johan-rocha

  committed
  4723727

• fix: transform refact

  johan-rocha

  committed
  b8a49e6

• fix: refact

  johan-rocha

  committed
  4597af7

• feat(#67): run all scripts

  johan-rocha

  committed
  a2d0a2b

• docs: outputs

  johan-rocha

  committed
  72723c4

• Merge branch 'develop' into feature-#1/Inep-Data-Scraper

  rafgpereira

  authored
  cb82285

• Merge pull request #70 from unb-mds/feature-#1/Inep-Data-Scraper

  Show description for b556943
  rafgpereira

  authored
  b556943

• Merge branch 'main' into develop

  rafgpereira

  authored
  4533518