Johan de Koning
Last week I bought a nice gadget at our discount supermarket (Aldi). The gadget is called a Digital notepad and makes it possible to record your notes while writing them down on paper. Your writing is stored on the internal memory as vector data. There is a lot of software supplied, but using them is not that straightforward.
Most of the software supplied with this gadget wants to convert the note to clear text. But because of my bad handwriting, most of the time the converted text makes no sense. I also want to keep the drawing which I added to my notes. So instead of converting it to text, I would be happy if I can archive the notes and make them searchable.
Microsoft OneNote gives Tablet PC users the option to add handwritten notes to there OneNote notebook. Those notes are made searchable, while the handwritten note still exists. Instead of using a Tablet PC I want to convert my Digital Notepad content to OneNote handwritten notes. And therefore I developed a small OneNote addin which makes this possible. The source code together with the installer can be found at http://top2onenote.codeplex.com. Within this blogpost, I will explain some of the steps I took to make this addin possible.
Reading a TOP file
There is less information available about the file format that is generated by the Digital Notepad. The file extension is called .Top and it looks like it is a format invented by Waltop International Corp. When searching on the Internet for more information I found this blogpost together with some perl and python scripts to convert a Top file to SVG. I will use the logic of these scripts to convert my notes to OneNote.
Develop an OneNote AddIn
Visual Studio 2008 gives you the possiblity to develop Office Add Ins. There are different project templates supplied for Word, Excel and Outlook… but not for OneNote. There are no Visual Studio templates available for OneNote, so developing a AddIn is not that easy. Again Google gives me a nice search result, a blogpost about creating a toolbar addin for OneNote 2007: http://blogs.msdn.com/descapa/archive/2006/08/31/734298.aspx. There is a step by step tutorial supplied which will explain the different steps to develop a toolbar Addin which will popup a Hello world message. The AddIn will make use of COM Interop, which makes it possible to use C#.
The interface which should be implemented does only contains two methods. I found out that using this interface will only gives you the active OneNote page. With the OneNote.ApplicationClass (part of the Microsoft.Office.OneNote.Interop assembly) you will update the OneNote page outside OneNote. This means that the functionality of your addin does not run within OneNote, but inside a DllHost container.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
using OneNote = Microsoft.Office.Interop.OneNote;
namespace Top2OneNoteAddIn
{
[ComImport, Guid("C9590FA7-2132-47fb-9A78-AF0BF19AF4E6")]
public interface IOneNoteAddIn
{
bool OnClick([In] String strActivePageID);
bool OnEvent([In] OneNote.OneNoteAddIn_Event evt, [In] String strParameter);
}
}
Note: together with OneNote 2007 I also installed the OneNote 2010 Beta. It seems like the AddIn functionality will change, because the addin is shown on the toolbar but clicking on it does not trigger the functionality. I think that with the release of OneNote 2010 that I will be easier to develop OneNote Addins, probably with the use of a Visual Studio template.
OpenFileDialog issue
The AddIn should, when clicked, open up a file dialog where the user can select the .top file which should be imported. Because the plugin does not run for within OneNote, the OpenFileDialog was shown below the OneNote window. I found a solution by supplying the Windows handler of OneNote as owner of the OpenFileDialog. This way the dialog is shown on top of OneNote
Process[] procs = Process.GetProcessesByName("OneNote");
IntPtr hwnd = procs[0].MainWindowHandle;
OpenFileDialog dialog = new OpenFileDialog();
dialog.Filter = "top files (*.top)|*.top|All files (*.*)|*.*";
dialog.Title = "Select a top file";
dialog.ShowDialog(new WindowWrapper(hwnd));
A little WindowWrapper class is added to the solution, which implements the IWin32Window interface making it possible to supply the window handler as argument to the ShowDialog method.
public class WindowWrapper : System.Windows.Forms.IWin32Window
{
private IntPtr _hwnd;
public WindowWrapper(IntPtr handle)
{
_hwnd = handle;
}
#region IWin32Window Members
public IntPtr Handle
{
get { return _hwnd; }
}
#endregion
}
InkAnalyzer
The vector data from the top file is converted to a collection of Stroke objects (for which the constructor gets a StylusPointCollection as argument). Those stoke objects are set to the InkAnalyzer. The Analyze method performs layout analysis, writing and drawing classification, and handwriting recognition. The result is given back as an AnalysisStatus object which will be used for building the XML structure used to update the OneNote page.
InkAnalyzer analyzer = new InkAnalyzer();
analyzer.AddStroke(stroke);
AnalysisStatus status = analyzer.Analyze();
if (status.Successful)
{
Console.WriteLine(analyzer.GetRecognizedString());
}
The AddStroke method has overloaded method for which you can specify the languageId of the stroke. By default the locale settings of the active thread will be used. In my case the Dutch language id was used (because of my region settings). I use English as my OS display language. When the Dutch language pack is not installed, the result of the analyzer will not be successful. So keep in mind to check if the correct language packs are installed to make handwritten recognition possible. The method analyzer.GetInkRecognizersByPriority() gives back a list of available InkRecognizers for the different languages.
Building the OneNote XML
You can update OneNote pages by providing a XML structure. Adding or updating depends if objectids are supplied. If this is the case, the object will be updated. An example of the XML used to add the handwritten notes to OneNote
<?xml version="1.0"?>
<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2007/onenote"
ID="{C6CCDE4D-7F47-46FA-9DD6-0A4CEC503E86}{1}{B0}">
<one:Outline>
<one:Position x="209.9905395507812" y="126.0"/>
<one:Size width="245.2818908691406" height="125.3196716308594"/>
<one:OEChildren>
<one:OE>
<one:InkWord recognizedText="hallo">
<one:Data>AMIGHQSQBOYCAYABKgAaH4tCcEWEdYC+0EXZBFjPVIrml8VPjwb4utLhmyK/
7HHMzCKxQYkOuxMs7nqh3rsTi+Ves0icm6ElWrdREvqHlTLy7BdBuMAWv/Mr9HrEZm2nDbJYSZodTkA0BmwQRVOAL
HoY1USZZUo0BjFbMxSVQJ6SzJhLtWqQHBLFWlUDC0gURSNGI1cNAAAABQcLZU9mZ2hpGRQyCACAHgIh4uJBMwgA
4BICSvPiQRGrqtNBZAMVRgBAShs20YdmzeA0YdmxgA6YdmsBow7GLAB0w7Nm8AAATxMEawMVRgBAahA+gEXPAA
ARXQADggAAIqIAAAqKAW+C/gdb+B19kqxUublEssCwCSiblSgFlABLLFSyypYWLLFlliiC/gR7+BH7ACTZc3LJqblks2W
CzZQ2VNliyklZSMkzVmrKsVKSgNyygAoAESBQIxrXX3TKARMEEgARIKFVgU7sSj5OjNx+XqrpdDQFFEYAQAAABRRGA
QAAAAoAESDcOupAmua8QApiMYL+CJP4IlAAAEoAAEoAgv4BG/gEfKycSbsm5di0qry2bEGy8tUKABEg4IA52V90ygE
TBBIAESCouC8UZctFT6fx0sc+lOXuBRRGAEAAAAUURgFAAAAKP0Aj06oIUjE4DWAKWiGC/gnj+CeQBBZZQACC/gEj
+ARWM5yUt25ddnaFiwAKABEgIH932l90ygETBBIAESAf+viILJADTKojJ+vOFaLHBRRGAEAAAAUURgGAAAAJFiKA30M
Qcv/ZAAp0SYL+BFP4EVAACUALAkJRU3NypSbCwUCC/gDT+ANeJMktzPHc7Le7dtVaNy7WOIgkuJbmpZGrZQAKAB
EgYAtv1V90ygETBBIAESAUVWvkl8H9TbQgjxGuoKlABRRGAEAAAAUURgDAAAAKP0Ai63IF0ksIC+AKggFlgv4MK/gw
25SpYsFlhLAESwlsmybLNypbKAssqWUlllixZYsAgv4EY/gRkAlgZYWWFlEs2WaFUTZVkqJUuUlgAssssVAACgARIGBI
qdtfdMoBEwQSABEgk453gdolgkGItwM1/odLkQUURgBAAAAFFEYBwAAACgARIGLKJkG7kLtA
</one:Data>
</one:InkWord>
</one:OE>
</one:OEChildren>
</one:Outline>
<one:Outline>
<one:Position x="225.0141754150391" y="461.2535400390625"/>
<one:Size width="680.25830078125" height="80.7023696899414"/>
<one:OEChildren>
<one:OE>
<one:InkWord recognizedText="dit">...
</one:Data>
</one:InkWord>
<one:InkWord recognizedText="is">
<one:Data>...
</one:Data>
</one:InkWord>
<one:InkWord recognizedText="een">
<one:Data>...</one:Data>
</one:InkWord>
<one:InkWord recognizedText="test">
<one:Data>...
</one:Data>
</one:InkWord>
</one:OE>
</one:OEChildren>
</one:Outline>
</one:Page>
Only the data for the first word is shown in this example to keep things clear. At the root element (Page) the ID of the existing page (provided as argument of the OnClick method) to define that the content should be inserted to the page.The InkWord data are stored as Ink Serialized Format with BASE64 encoding and placed together with the recognizedText to make searching within the handwritten notes possible.
InkWordNode iwNode = node as InkWordNode;
using (MemoryStream ms = new MemoryStream())
{
iwNode.Strokes.Save(ms);
byte[] isfBytes = ms.ToArray();
dataNode.SetValue(Convert.ToBase64String(isfBytes));
}
Updating OneNote
Updating the OneNote page is done by creating an instance of the OneNote.ApplicationClass. The UpdatePageContent method is used whereby the created XML structure is supplied as argument. There is a little delay between executing the importer and when the page is updated. The cause of this delay is probally that the OneNote page is updated outside OneNote and that it will be triggered to refresh the page content.
OneNote.ApplicationClass onApplication = new OneNote.ApplicationClass();
XNamespace one = "http://schemas.microsoft.com/office/onenote/2007/onenote";
XElement root = new XElement(one + "Page", new XAttribute(XNamespace.Xmlns + "one", one));
root.Add(new XAttribute("ID", strActivePageID));
BuildXML(analyzer.RootNode, root, one);
onApplication.UpdatePageContent(root.ToString(), DateTime.MinValue);
What’s next?
The result of the importer is the same when writing the text directly with the use of a Tablet PC. Probably the results of a Tablet PC are maybe even more secure, but that’s also the reason that I don’t want to convert the handwritten notes to text.
Search within a handwritten note
The importer could be optimized in different ways. Some issues/ideas to improve this add in
- Language selection – selected by import or set as a preference
- Insert at cursor position – currently the location is set to the top left. It is possible to retrieve the cursor position by requesting the page content (GetPageContent) with the use of the OneNote.ApplicationClass. The location of the cursor is marked as a selection inside the retrieved XML structure
- Improve text recognition – the InkAnalyzer has different properties to improve the text recognition. This way better results are achieved
- Drag and drop support – instead of using the open file dialog, it would be nice if you can just drag the top file on your OneNote page
Like I mentioned, you can find the source at http://top2onenote.codeplex.com. It is not my goal to put a lot of effort in the development of this AddIn. If you are interested in improving this AddIn, send some feedback and I can add you as contributor to this project. If you have other questions, ask them and I will try to answer them.
