"The easiest way to think about Memex is: How can I make the unseen seen?"
This time last year Pablo and I were chatting about the previous year and what to expect in 2013. I noticed that our team had almost doubled in size in the previous year and we wondered could that possibly continue in 2013?
MongoDB was used early on at Scrapinghub to store scraped data because it's convenient. Scraped data is represented as (possibly nested) records which can be serialized to JSON. The schema is not known ahead of time and may change from one job to the next. We need to support browsing, querying and downloading the stored data. This was very easy to implement using MongoDB (easier than the...
It's finally time to start a Scrapinghub blog! In the upcoming months we expect to open our private beta to new customers, launch new services, add many new features and continue to contribute to open source projects. It's about time we had a way to to tell everyone about all the great things that are happening!