Crawly.DataStorage (Crawly v0.12.0) View Source

Data Storage, is a module responsible for storing crawled items. On the high level it's possible to represent the architecture of items storage this way:

           ┌──────────────────┐
           │                  │             ┌------------------┐
           │   DataStorage    <─────────────┤ From crawlers1,2 │
           │                  │             └------------------┘
           └─────────┬────────┘
                     │
                     │
                     │
                     │
        ┌────────────▼─────────────────┐
        │                              │
        │                              │
        │                              │

┌───────────▼──────────┐ ┌───────────▼──────────┐ │ DataStorageWorker1 │ │ DataStorageWorker2 │ │ (Crawler1) │ │ (Crawler2) │ └──────────────────────┘ └──────────────────────┘

Link to this section Summary

Functions

Returns a specification to start this module under a supervisor.

Callback implementation for GenServer.init/1.

Link to this section Functions

Returns a specification to start this module under a supervisor.

See Supervisor.

Callback implementation for GenServer.init/1.

Link to this function

start_worker(spider_name)

View Source

Specs

store(atom(), map()) :: :ok