A development blog for NeoMem, an information manager.
There should be a new release towards the end of the month - I’m working on tracking down some bugs, adding export (and possibly import), and packing my apartment (moving from Houston to Austin at the end of the month).
I can’t wait until books are all digital - I have so many, and they take up so much space…
All it will take is a more readable display (like the electronic ink displays that look more like paper), that’s thin and light. And some nice 3d animation of turning pages…
Tags: No Tags
Happy New Year 
Here’s a shot of the new website, under development. No snazzy graphics yet. And probably too many menu options on the left at the moment (less is more, as they say). It’s based on what will eventually be a generic project management tool, called neomemProject.
I’m hoping to have this take over this site in February or March. We’ll see how that goes…
NeoMem has been turning into various subprojects over the last several months, and now they have names -
I’ll be working on all of this stuff over the coming year. Once I have more of the skeleton of neomemRails working, I’ll start seeking some more developers to help out - that might be a couple of months.
I recently fixed a couple of bugs in neomem, and added one feature (using an .ini file instead of the registry). I’ll be posting these updates as I go, but I don’t have an automated deployment system set up yet, so it’s sort of slow going at the moment.
Also slowly catching up on emails and forum posts and blog comments…
Tags: No Tags
Well a couple more bug reports came in - I’m going to have to bite the bullet and start tracking these things down. And while I’m at it I’ll start on the export and import. Probably just csv/tab delimited for now, but I’ll look into Keypad and Treenote also, see how involved they would be. I’ll aim for a new release in January sometime.
There are a couple of Rails and Subversion enabled webhosts that look promising - A Small Orange and OCS Solutions. So hopefully soon I’ll have this site on Rails, and the source code in Subversion, online.
Also just converted the old CVS repository to Subversion.
Update: I signed up with OCS Solutions, and am switching the site over to it.
Tags: No Tags
Aside from dealing with preparing to move in January and other things, I’ve been playing around with the Ruby on Rails interface. It might wind up being all I need, for a while. And the Ruby classes that wrap the database could be used by other interfaces also, eg a Ruby/wxWidgets standalone program, or a console-based application/view.
Some stumbling blocks though -
I’d assumed it would be pretty easy to synchronize information to a website, ie you have your local MySQL database and one on your website, and just click a button and have them synchronized. Well, maybe if you pay a lot of money for a program to do that. I don’t know if there are any open source programs that will do that. (update: Daffodil Replicator does). MySQL does have replication, but it’s only in one direction. Maybe other databases are more advanced in this area? The concept is pretty simple - surely somewhere out there is code to do this? If not I’ll have to separate out my NeoMem related objects and import them to my webhosts’s database, so I can start hosting this site via Rails.
Which leads to another issue - finding a webhost that will let you host a (possibly unstable) Rails application. My current host (1and1.com) doesn’t seem to support Rails. So I’m looking into others. And it’s not fun. eg this discussion. Hosting Subversion repositories would be essential also.
Another problem is that web hosts don’t like to give you direct access to your database - you usually only have access it through a web interface, like phpMyAdmin. This was (not welcome) news to me. Though programs like Navicat or SQLYog have tunnels to let you access your database through the normal routes. But they cost money.
Of course, all of this would be moot if I had my own webserver. Which is not something I am prepared to deal with at this point (or ever, really), and obviously should not be a requirement for using NeoMem to run your website. The complexity involved with running a Rails application does mean there would be an opportunity for a small business though, to host users’ data, say on the neomem.com domain. But that’s not my focus at this point.
So those things are throwing a few wrenches into my plans to have this site being driven off of a database. It’s still the goal, but I don’t know how long it will take to reach it.
Anyway, that’s where things stand at the moment. I really would like to get the source code for NeoMem 1.1 hosted online this month - I might wind up just throwing it into SourceForge if I can’t find a decent webhost for now. And yes, export and import and bugfixes are high priorities - hopefully I’ll be able to at least start on some of that this month.
Tags: No Tags
The browser interface is coming along faster than expected, thanks to Ruby on Rails - one of the best thought-out pieces of software I’ve ever worked with. Ruby itself is a great language, and the infrastructure around it is much nicer than Python (eg install programs from the command line by “gems install rails” - it’ll search it out on the internet, download it, and install it for you).
Here’s the current Home View - basically a table of contents on your data, with a listing of all the Types stored, some statistics, etc. I’m still working out how the pages will be layed out. There might be a list of views along the top - Home, List, Code, Calendar, Schedule, Project, etc. And some standard filter, sort, and group controls.
Today I’m working on a Project View, which will summarize a project and its subprojects, and will eventually be used for this site. I’ve given up on using any of the existing bug tracking programs - it doesn’t seem worth the effort of cramming my data into their format, when I can get a simple system up and running on Ruby on Rails, all tied into the rest of the project information (goals, related projects, documentation, people, etc.).
Tags: No Tags
A note on the synchronization issue - I decided to store a GUID in each record, so that if someone adds a new record to your website data, when you synchronize with your local data (which might have the same integer id key), the code can check if it’s actually the same record, and renumber one of them if not (and any related records). That way it can still just use an integer for the key field.
There was some discussion about using GUIDs as primary keys for Rails, but ugh, that would not be pleasant…
Tags: No Tags
One thing I’d really like NeoMem to do is be able to publish your information to a website. And then keep the website in synch with your local datastore, if people edit or add information there. Here’s a simplified diagram of what it’ll look like…
I figure it would be best if I concentrated on the browser interface at the moment, since that will be the most useful, eg for publishing all the project-related information for NeoMem to this site. And it would also function as the user interface on the local data store, until the new Ruby-based interface gets up to speed. The browser interface will probably be based on Ruby on Rails, or possibly Nitro. We’ll see how that goes.
I’ve been avoiding the old C++ code - it’s really not a pleasant thing to deal with, and tracking down old bugs is just not what I want to deal with at the moment (massive time drain). I’d rather just get the “Export All to CSV” working, and then migrate all my data to the new version. Abandon ship! If there’s enough interest in having a really light, C++ based UI on the data then the old program could be fixed up, migrated to wxWidgets and made cross-platform. It might be useful, for a PDA or something.
So the current order of action is:
Sorry to be leaving all those bugs in there, for now… I just don’t seem to have the time or energy to deal with them, and maybe it’s better that I concentrate on the new version anyway… My lease is up at the end of January, and I’ll probably be moving to Austin - I’m planning to digitize a lot of the things in my apartment (file cabinets, notebooks, artwork, etc), with a digital camera mounted on a 2×4, so I won’t have to move so much stuff. The nine bookshelves of books will have to wait a while for their turn, unfortunately…
Tags: No Tags
Here’s is a possible idea for the backend - the objects would be crammed into a relational database for now (unless or until there are good open source oodb’s available). A simple synchronize command would generate a file for each object (stored in YAML code or something similar). Desktop Search plugins would be indexed off of these files, and they could also be edited in Notepad and then resynched with the database.
Putting them in files would also allow storing their history in a repository like Subversion (not sure how easily extractible that history would be though, eg for field values). This would also allow use of standard merging tools for keeping the data in synch with other copies. For instance, if you have a giant database on your home machine, and then upload a chunk of it to a website (say all the objects marked as public) - then they could both be edited at the same time and kept in synch.
Eventually, I’d like to wrap all my existing files with metadata (eg by putting each file in a .zip archive along with a metadata text file (using yaml), and give it some standard extension, like .neoobj or something). These files would then sit in the folder along with all the other yaml object files, and the database would be kept in synch with their metadata also (leaving the actual binary file data on the file system). The net effect is that you would have a virtual file system sitting on top of the existing one, that is platform independent, open source, and provides metadata and history for all your files.
The Windows Explorer plugin would just be something that would expose the folder containing all these objects, and let you save files there (which would then wrap them in a metadata wrapper automatically, etc), or load them, etc. I’m not sure how this would work out yet, but it would be a way of letting legacy apps save files to your data store, instead of the old file system.
Note that if you wanted a portable version to take along with you, all you’d need is the SQLite database and a small frontend (eg the old NeoMem frontend, since it’s in C++ and makes for a small .exe). It could also be adapted for a PDA someday (which I would really like to have, to take the place of all these index cards I’m always carrying around with me). More on that later…
Here’s a rough picture of the architecture:
Developed with Graphviz (source for the drawing here).
Tags: No Tags
Here’s the latest feature/bug list: NeoMem Feature List (94kb pdf).
The next release (1.1d) will be just bug fixes (I know, not very exciting, but important). Then I’ll try and get to import/export and a few other requested features for 1.2. But I’m trying to balance my time with working on the new design as well, so some of those things might take a while. I might wind up cutting down the number of features for 1.2 in order to get import/export out. It’ll also help to have the code hosted on this site - that’s coming soon…
Note that version 1.99 is just a catchall for all future features. And 2.0 is for features that will probably best be implemented under the new architecture. And also, this still does not have a bunch of features that are scattered throughout my own notes - those will be added to the database gradually. But it incorporates everything from email, the forum, and the blog, assuming I didn’t miss anything.
And my installation of Access 97 still won’t export to HTML, despite reinstalling it, hence the pdf. I’ve been playing around with Ruby on Rails and MySQL a bit, so eventually this list will be interactive. Either that, or I’ll just put the project into Trac for the time being.
Tags: No Tags
ah well, life happens, and things get in the way - i’ve already gotten behind schedule - just a lot of distractions this last week. i’ve still been working on the database though, which feeds into ideas for the backend. and as usual, lots of other ideas occur along the way. they get jotted down on index cards, and eventually make their way into the database. and there’s a large backlog of blog entries as well.
anyway, what i’m doing is trying to cram all of my information into this (relational) database. currently the weak spots are
i’m sure somebody, somewhere, has solved these problems before. there are some pretty advanced commercial databases out there. sure would be nice to be able to use one of those for the backend…
multiple field values - eg type=”person, performer”, date=”1789 or 1794″, date=”oct 1994 to dec 1998″, filter=”apples and oranges but not lemons”. right now i’m just storing these in the database just like that (ie text). presumably a parser could come along and store the information “properly”, though i’m not sure yet what form that will take. neomem had code to handle multiple item links, but it really needs to be expanded to be able to store arbitrary types of information with arbitrary relationships (ie not just AND).
composite field types - most databases are way too limited in their field types, so much so as to be almost useless - i’m just storing everything as text for now. but basically, all types are composite types, made up of smaller bits of information (until you get to numbers and strings). eg a person is made up of name, address, home phone, etc. an address is made up of street, city, state, zip, etc. a weight is made up of a number and a unit (eg “150 lbs”). so it would be nice to handle all of this generically, to arbitrary depths. right now neomem makes a distinction between object types and field types, but that’s pretty arbitrary (and based in part on the distinction between records and fields, which again is somewhat arbitrary). i think there are some advanced databases that let you define composite types like this, though i haven’t yet looked for any open source projects.
type parsers - ideally, each type would have its own parser, which might call subparsers. eg a NumberUnit parser would call a Number parser to interpret “150″ and a Unit parser to interpret “lbs” (the whole point being to do calculations and convert between units automatically, eventually). and dates, good grief - the number of time that wheel has been reinvented. it really should be a service of the operating system, ie pass it some text and it’ll interpret it for you. well i guess most operating systems do have such parsers but they’re nowhere near as powerful as they could be - hence, they’re basically useless. neomem’s current date parser is a step in that direction though. and all fields should also allow ranges, multiple values, and qualifiers as well, eg “jan 15, 1970 to summer 1975?”.
relationships - there are a lot of different ways of handling this, and i haven’t figured out the best way to do it yet. in some cases all you really need is to store one way information (eg .type=book), since to go the other way you can just do a search (currently how neomem handles relationships). sometimes you’d like two way storage (eg lotr.author=tolkein, tolkein.books=lotr,silmarillion) - and doing it automatically would be nice (ie you add it one place and the db adds the other side). and sometimes you’d like to attach information to the relationship (ie treat the relationship like any other object). ultimately it would be nice to have all relationships stored explicitly as objects - it would make things like displaying the entire network of objects and their relationships easier. and also for attaching “weights” to relationships, eg for building up networks of related objects.
version history - it would be nice if the database tracked the history of any field. this would include the large text fields also, allowing you to track back through the changes. and the information should be easily accessible, for viewing, editing, or feeding into plotting.
i really don’t want to reinvent any wheels though - i’m going to do some research and see what other databases are out there. hopefully there’ll be something useful…
Tags: No Tags