While playing with F# I found that I'm really missing intellisense support for FSI interactive. It is very useful for exploring APIs and language features. I know, "official" suggestion is to write code in script file where full intellisense is available, and then execute it in FSI. But that is less then optimal, at least for me.
Lack of intellisense is even more surprising given that console version of fsi.exe has some limited support for autocompletion. I even considered embedding console window with running FSI in Visual Studio, but that turned our to be problematic.
Then everything I expected to do it taking walkthrough and integrating it with FSI autocompletion APIs. Why should it take longer than couple of evenings? But as usual for software development nothing works as expected from the first attempt.
First I had to find a way to access those FSI APIs. FsiLanguageService is registered as a Visual Studio extensibility service. This is our entry point, everything else can be solved using Reflection. It is obviously a hack and it'll likely break if FSI sources are changed. But it will work, and telling from other's code this is the way extensibility is done for Visual Studio.
In order to access internal properties and fields using C# dynamic keyword we can use ExposedObject. Also I had to modify it a bit because it doesn't support fields declared in nested classes. It is a bit ugly, but does its job:
The second problem is that for some reason FSI is compiled with #FSI_SERVER_INTELLISENSE flag off. It complicates things a lot. There are 3 options here:
1. Recompile FSI with #FSI_SERVER_INTELLISENSE flag on and use modified fsi.exe file. I tried this, and it worked. Also I didn't like to use hacked FSI, so decided to move further.
2. Implement autocomplete APIs on my own.
3. Try to find an entry point into FSI process, and use reflection to call autocomplete APIs.
It wasn't clear if it is possible to find a way to get into existing FSI process, therefor I decided to start with my own implementation.
For me it meant that I need to run some code within FSI process, and that I need to establish some intra-process communication. First I tried to use Remoting since it is already used by FSI server, and it almost worked. Once in a while it failed with some cryptic errors which have no sense at all. So I replaced it by WCF service operating over NamedPipes, and it worked much better. Here is my service definition:
I didn't come up with anything better than adding reference to my dll and then executing command to start my server by piping those commands directly into FSI input. Not perfect solution, but works well.
let getVariableNames(fsiAssembly:Assembly) =
fsiAssembly.GetTypes()//FSI types have the name pattern FSI_####, where #### is the order in which they were created
|> Seq.filter (fun ty -> ty.Name.StartsWith("FSI_"))
|> Seq.sortBy (fun ty -> ty.Name.Split('_').[1] |> int)
|> Seq.collect (fun ty -> getPropertyInfosForType(ty))
|> Seq.map (fun pi -> pi.Name)
|> Seq.distinct
I also added namespaces opened in F# by default- Microsoft.FSharp.Collections and Microsoft.FSharp.Core.Printf. It helped a bit with usability, but there was another problem.
Autocomplete popup just rejected processing arrow down button. It is critical for navigation buttons to work for autocomplete popup, so I couldn't leave this issue unsolved. One of the biggest problems with Visual Studio extensibility is lack of good documentation. So the best source of information for me were GitHub sources for similar projects, and debugger. It turned out that CommandHandlers for key events in Visual Studio are organized in a hierarchical manner. If top level CommandHandler processes key event, and marks event as processed, there is nothing we can do on lower level CommandHanlder. In case with FSI window, fsiToolWindow processing arrow up/down keypress events for navigating through history, and therefore intercepted those events. And in fact when cursor was not set to the bottom line of FSI those events worked just fine since FSI doesn't process them for navigating history. I guess I tried every reasonable approach to make it work, but it didn't help. So the only way for me to solve it was to make FSI think that it should not iterate over history. Luckily there is a way:
Finally everything worked, but after playing with this solution for some time I realized that it is far from perfect.
So I got back to original idea - find a way to hack into running FSI process in order to get access to autocomplete APIs. It is possible to use reflection in order to explore type system of a running process, but we need an entry point in the process - reference to some variable or static field. So I carefully studied FSI source code, but didn't find anything I could use for that purpose.
I tried hard to find a way to find a reference to the class or variable of the running process given that I know that there in only one instance of that class exists in a process. It turned out impossible, or at least I didn't find any reasonable approach to do that. Another approach I investigated is hacking running process itself. There are ways to do that using CLR Profiler APIs, but that is a bit too extreme to be usable. After struggling with this for a while, I've got an idea. The fact that I didn't find any "entry" points into FSI process, doesn't mean that they don't exist. And if they exist, I should be able to see them in memory profiler. So I downloaded CLR profiler, and references view gave me the answer. System.AppDomain is a globally accessible by System.AppDomain.CurrentDomain, and I know that there is only one subscriber to _unhandledException event. So this is our entry point.
Here is how code for executing autocomplete API on a running FSI process end up looking:
Extremely hacky, but not too complicated once you know what to do.
Since this API doesn't tell what kind of completion it returns (variable, method, class, etc.), I merged its results with output of my home grown provider in order to be able to show icons.
So finally everything worked. Intellisense support is still far from perfect, but much better than nothing. I guess I'll try to integrate it with https://github.com/fsharp/FsAutoComplete later, we'll see.
As a result I've finally got decent Intellisense support for FSI, but it took 10-20 times of my original estimate to get there. Not sure that it was worth of time spent, but it was definitely interesting and unconventional project, so I had some fun while building it. Hope anyone will find it useful. Here is a link to GitHub project: https://github.com/vlasenkoalexey/FSharp.Interactive.Intellisense
Idea to create a dev blog crossed my mind long time ago, but I didn't get to finally do it until now. Hope someone will find my thoughts and experience useful.
Today I want to share my experience of integrating FSharp.Data library into Windows Store application. It took couple of evenings to finally make it work, and few times I thought about dropping this idea and doing everything in good old C# instead. Glad that it finally worked out. So I'd like to share my findings in case someone wants to build something similar.
I started to learn F# in 2011, and the more I got to know it, the more I liked it. Since then I'm trying to use it whenever possible, but unfortunately can't say that I had a lot of success with that.
Recently I decided to rewrite part of Window Store application I develop as a hobby project, and decided to give F# a try here. Application has HTML parsing component implemented in C# using HTML Agility Pack. Given that there is an amazing FSharp.Data library with HTML provider, it looked promising to rewrite that code in F#.
My first disappointment was that Windows Store does not support F# natively. I couldn't believe that Microsoft allows building Windows Store apps using C++ and JavaScript, but at the same time F# code needs to go into portable class library (PCL). Regardless of all statements about growing F# share and predictions of its bright future this fact makes it look like a niche language in .NET family. It looks even more strange taking into account that Apple have chosen to go functional with Swift, but that is another story. Hope that Microsoft will start taking F# more seriously in the future.
Luckily there is a great blog post describing how to integrate F# into Windows Store application: http://ps-a.blogspot.com/2013/01/windows-store-apps-with-f-part-1-make.html. It helped to setup project and understand how all pieces work together. In the first approach I tried to take maximum advantage out of F# and attempted to reimplement ViewModel. It didn't work nicely. First problem was that ViewModel concept is all about state and .NET events. All that code looks clumsy in F#. Second problem was that I have quite a bit of helper code and ViewModel base class in C# project. It couldn't be referenced from F# unless that code is moved to another C# portable library. That option didn't appeal to me much, so I decided to drop this idea. It might work better in case all ViewModels and related logic in the project are implemented in F#. In fact http://reactiveui.net/ looks like a perfect match here. Unfortunately there is no information available describing how it works with F#. I plan to try it out the next time I build Windows Store app.
So, eventually I decided to keep ViewModel in C# and only port library responsible for HTML parsing and data processing to F#. First I had to figure out how to use FSharp.Data APIs from F# interactive. I installed nuget package, and started exploring.
Install-Package FSharp.Data
It turned out that nuget package contains 2 folders: net40 and portable-net40+sl5+wp8+win8.
net40 is used for native .NET applications, and portable-net40+sl5+wp8+win8 for platforms which work with portable class library including Windows 8. Therefore we'll refer to
packages\FSharp.Data.2.1.0\lib\net40\FSharp.Data.dll in F# interactive and add reference to packages\FSharp.Data.2.1.0\lib\portable-net40+sl5+wp8+win8\FSharp.Data.dll in Windows Store project.
Ok, it looks like provider works, but it doesn't do very good job in providing strongly typed wrapper around HTML document structure. It exposes basic APIs for iterating over document nodes and doing matching using predicate function. This is just enough in most of cases.
Next I added sample module file, referenced F# project from C# Windows Store application and tried to compile it. It didn't work like that:
2>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\AppxPackage\Microsoft.AppXPackage.Targets(852,9): error MSB3816: Loading assembly "C:\Projects\FsharpWindowsStorePrototype\FsharpPortableLibrary\bin\Debug\FSharp.Data.dll" failed. System.IO.FileNotFoundException: Could not load file or assembly 'FSharp.Core, Version=2.3.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\AppxPackage\Microsoft.AppXPackage.Targets(852,9): error MSB3816: File name: 'FSharp.Core, Version=2.3.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
After some googling I found out that FSharp.Data doesn't support PCL profile 7. It turned out that there are multiple PCL profiles to target different set of platforms. I spent few hours trying to change profile type to 47. Solution was unexpected, but straightforward:
1. Use legacy Project Library template for .NET Framework 4.0
2. In project settings change target F# runtime to 3.0 (FSharp.Core 2.3.5.0)
Finally I could build solution and do HTTP requests ... almost.
It turned out that web server that hosts site that I'm scraping requires UserAgent to be set on HTTP request. There is a known issue with UserAgent header in Windows Store apps. It is just not possible to set UserAgent header on HttpWebRequest. FSharp.Data provides nice Http helper utility, and I had a hope that it would work:
1: openSystem2: openFSharp.Data3: openFSharp.Data.HttpRequestHeaders4: 5: // Run the HTTP web request6: Http.RequestString7: ( "https://www.google.com/search?q=super",
8: query= [ "q", "super" ],
9: headers= [ AcceptHttpContentTypes.Any; UserAgent"Mozilla/5.0" ])
namespace System
namespace FSharp
namespace FSharp.Data
And it does work ... but only in F# interactive which uses .NET version of the binary. I thought I'm doing something wrong until I found this code in FSharp.Data souces:
Multiple items
union case DataOrError.Data: String -> DataOrError
--------------------
namespace System.Data
type String =
class
new : char -> string
new : char * int * int -> string
new : System.SByte -> string
new : System.SByte * int * int -> string
new : System.SByte * int * int * System.Text.Encoding -> string
new : char [] * int * int -> string
new : char [] -> string
new : char * int -> string
member Chars : int -> char
member Clone : unit -> obj
member CompareTo : obj -> int
member CompareTo : string -> int
member Contains : string -> bool
member CopyTo : int * char [] * int * int -> unit
member EndsWith : string -> bool
member EndsWith : string * System.StringComparison -> bool
member EndsWith : string * bool * System.Globalization.CultureInfo -> bool
member Equals : obj -> bool
member Equals : string -> bool
member Equals : string * System.StringComparison -> bool
member GetEnumerator : unit -> System.CharEnumerator
member GetHashCode : unit -> int
member GetTypeCode : unit -> System.TypeCode
member IndexOf : char -> int
member IndexOf : string -> int
member IndexOf : char * int -> int
member IndexOf : string * int -> int
member IndexOf : string * System.StringComparison -> int
member IndexOf : char * int * int -> int
member IndexOf : string * int * int -> int
member IndexOf : string * int * System.StringComparison -> int
member IndexOf : string * int * int * System.StringComparison -> int
member IndexOfAny : char [] -> int
member IndexOfAny : char [] * int -> int
member IndexOfAny : char [] * int * int -> int
member Insert : int * string -> string
member IsNormalized : unit -> bool
member IsNormalized : System.Text.NormalizationForm -> bool
member LastIndexOf : char -> int
member LastIndexOf : string -> int
member LastIndexOf : char * int -> int
member LastIndexOf : string * int -> int
member LastIndexOf : string * System.StringComparison -> int
member LastIndexOf : char * int * int -> int
member LastIndexOf : string * int * int -> int
member LastIndexOf : string * int * System.StringComparison -> int
member LastIndexOf : string * int * int * System.StringComparison -> int
member LastIndexOfAny : char [] -> int
member LastIndexOfAny : char [] * int -> int
member LastIndexOfAny : char [] * int * int -> int
member Length : int
member Normalize : unit -> string
member Normalize : System.Text.NormalizationForm -> string
member PadLeft : int -> string
member PadLeft : int * char -> string
member PadRight : int -> string
member PadRight : int * char -> string
member Remove : int -> string
member Remove : int * int -> string
member Replace : char * char -> string
member Replace : string * string -> string
member Split : char [] -> string []
member Split : char [] * int -> string []
member Split : char [] * System.StringSplitOptions -> string []
member Split : string [] * System.StringSplitOptions -> string []
member Split : char [] * int * System.StringSplitOptions -> string []
member Split : string [] * int * System.StringSplitOptions -> string []
member StartsWith : string -> bool
member StartsWith : string * System.StringComparison -> bool
member StartsWith : string * bool * System.Globalization.CultureInfo -> bool
member Substring : int -> string
member Substring : int * int -> string
member ToCharArray : unit -> char []
member ToCharArray : int * int -> char []
member ToLower : unit -> string
member ToLower : System.Globalization.CultureInfo -> string
member ToLowerInvariant : unit -> string
member ToString : unit -> string
member ToString : System.IFormatProvider -> string
member ToUpper : unit -> string
member ToUpper : System.Globalization.CultureInfo -> string
member ToUpperInvariant : unit -> string
member Trim : unit -> string
member Trim : char [] -> string
member TrimEnd : char [] -> string
member TrimStart : char [] -> string
static val Empty : string
static member Compare : string * string -> int
static member Compare : string * string * bool -> int
static member Compare : string * string * System.StringComparison -> int
static member Compare : string * string * System.Globalization.CultureInfo * System.Globalization.CompareOptions -> int
static member Compare : string * string * bool * System.Globalization.CultureInfo -> int
static member Compare : string * int * string * int * int -> int
static member Compare : string * int * string * int * int * bool -> int
static member Compare : string * int * string * int * int * System.StringComparison -> int
static member Compare : string * int * string * int * int * bool * System.Globalization.CultureInfo -> int
static member Compare : string * int * string * int * int * System.Globalization.CultureInfo * System.Globalization.CompareOptions -> int
static member CompareOrdinal : string * string -> int
static member CompareOrdinal : string * int * string * int * int -> int
static member Concat : obj -> string
static member Concat : obj [] -> string
static member Concat<'T> : System.Collections.Generic.IEnumerable<'T> -> string
static member Concat : System.Collections.Generic.IEnumerable<string> -> string
static member Concat : string [] -> string
static member Concat : obj * obj -> string
static member Concat : string * string -> string
static member Concat : obj * obj * obj -> string
static member Concat : string * string * string -> string
static member Concat : obj * obj * obj * obj -> string
static member Concat : string * string * string * string -> string
static member Copy : string -> string
static member Equals : string * string -> bool
static member Equals : string * string * System.StringComparison -> bool
static member Format : string * obj -> string
static member Format : string * obj [] -> string
static member Format : string * obj * obj -> string
static member Format : System.IFormatProvider * string * obj [] -> string
static member Format : string * obj * obj * obj -> string
static member Intern : string -> string
static member IsInterned : string -> string
static member IsNullOrEmpty : string -> bool
static member IsNullOrWhiteSpace : string -> bool
static member Join : string * string [] -> string
static member Join : string * obj [] -> string
static member Join<'T> : string * System.Collections.Generic.IEnumerable<'T> -> string
static member Join : string * System.Collections.Generic.IEnumerable<string> -> string
static member Join : string * string [] * int * int -> string
end
type Debug =
class
static member Assert : bool -> unit
static member Assert : bool * string -> unit
static member Assert : bool * string * string -> unit
static member Assert : bool * string * string * obj [] -> unit
static member AutoFlush : bool with get, set
static member Close : unit -> unit
static member Fail : string -> unit
static member Fail : string * string -> unit
static member Flush : unit -> unit
static member Indent : unit -> unit
static member IndentLevel : int with get, set
static member IndentSize : int with get, set
static member Listeners : System.Diagnostics.TraceListenerCollection
static member Print : string -> unit
static member Print : string * obj [] -> unit
static member Unindent : unit -> unit
static member Write : string -> unit
static member Write : obj -> unit
static member Write : string * string -> unit
static member Write : obj * string -> unit
static member WriteIf : bool * string -> unit
static member WriteIf : bool * obj -> unit
static member WriteIf : bool * string * string -> unit
static member WriteIf : bool * obj * string -> unit
static member WriteLine : string -> unit
static member WriteLine : obj -> unit
static member WriteLine : string * string -> unit
static member WriteLine : obj * string -> unit
static member WriteLine : string * obj [] -> unit
static member WriteLineIf : bool * string -> unit
static member WriteLineIf : bool * obj -> unit
static member WriteLineIf : bool * string * string -> unit
static member WriteLineIf : bool * obj * string -> unit
end
Full name: System.Diagnostics.Debug
Debug.WriteLine(value: obj) : unit
Debug.WriteLine(message: string) : unit
Debug.WriteLine(format: string, args: obj []) : unit
Debug.WriteLine(value: obj, category: string) : unit
Debug.WriteLine(message: string, category: string) : unit
Object.ToString() : string
type Exception =
class
new : unit -> System.Exception
new : string -> System.Exception
new : string * System.Exception -> System.Exception
member Data : System.Collections.IDictionary
member GetBaseException : unit -> System.Exception
member GetObjectData : System.Runtime.Serialization.SerializationInfo * System.Runtime.Serialization.StreamingContext -> unit
member GetType : unit -> System.Type
member HResult : int with get, set
member HelpLink : string with get, set
member InnerException : System.Exception
member Message : string
member Source : string with get, set
member StackTrace : string
member TargetSite : System.Reflection.MethodBase
member ToString : unit -> string
end
Note that there is an bug in FSharp.Data 2.1.0, so that any closing tag with number like </1> breaks the parser.
Finally everything worked, but can't say that it was the end of the story.
When I tried to pull project from Git to another computer it refused to build. For some reason F# project was broken. Nuget pulled all dependencies, but for build didn't work.
Googling and experimenting proved that there is an issue with Microsoft.Bcl dependency which is required by Microsoft.Net.Http. By default Nuget tries to pull older version of this library, while Microsoft.Net.Http works only with the most recent one. The solution which worked for me was to clean Nuget cache, remove Microsoft.Bcl and Microsoft.Bcl.Build projects from packages folder, and request Nuget to install those projects manually before installing Microsoft.Net.Http.
Install-Package Microsoft.Bcl
It helped to solve dependency issues.
When I did that recently, I also updated FSharp.Data to version 2.2.0. This update broke my application, and now I have to investigate what changed.
Was it worth messing up with F#? Definitely I've spent way more time trying to fix all issues on the way than I'd spend writing same logic in C#. But it was fun and I'm glad that finally everything worked out.
