University of Maryland

Program Structure

DICE is written exclusively in Python. It relies heavily on assumptions about what files for your web scrapers are called in order to keep needless configuration to a minimum.

 

structure

Every database should contain a download.py and an extract.py. All steps that follow are generic enough that no further part of the process should be database-specific. You can add in whatever other files you want. It is sometimes convenient to add in a file containg a class that encapsulates what an article looks like and share its use between download.py and extract.py.