This is a very informal blog of the Cabal hacking team at OdHac: Sasha Manzyuk, Bram Schuur and me (Dominic Steinitz). I really enjoyed pair programming with both of them and I certainly wouldn’t have tracked down the bug I investigated without their help.
Also a big thank you to Roman Cheplyaka for organising a great event.
Cabal Bug Fixing I
The problem is indeed in the function Distribution.Simple.Utils.wrapText, which is used to wrap messages at 79 characters. No attempt is made to detect whether a space at which a line is wrapped is part of a file name. One can reproduce the problem more easily as follows:
$ cabal install http://hackage.haskell.org/packages/archive/groups/0.2.0.1/groups-0.2.0.1.tar.gz --prefix "/home/manzyuk/foo/bar/baz quux"
Downloading
http://hackage.haskell.org/packages/archive/groups/0.2.0.1/groups-0.2.0.1.tar.gz
Resolving dependencies...
In order, the following will be installed:
groups-0.2.0.1 (reinstall)
Warning: Note that reinstalls are always dangerous. Continuing anyway...
Configuring groups-0.2.0.1...
Building groups-0.2.0.1...
Preprocessing library groups-0.2.0.1...
[1 of 1] Compiling Data.Group ( src/Data/Group.hs, dist/build/Data/Group.o )
In-place registering groups-0.2.0.1...
Installing library in /home/manzyuk/foo/bar/baz
quux/lib/groups-0.2.0.1/ghc-7.4.1
Registering groups-0.2.0.1...
Installed groups-0.2.0.1Note that the line starting with “Installing library in” is wrapped at the space that is part of a directory name.
I suppose, a proper way to fix it would be to use an algebraic data type rather than String to represent messages, something like Text.PrettyPrint.Doc, so that file names can be treated as unbreakable entities that shouldn’t be wrapped. This, however, seems to require quite some re-engineering of error handling in Cabal. Should I go ahead and do it?
2013-05-04 Sat: Mikhail Glushenkov commented: “I think it’s a minor issue and your efforts are better spent elsewhere.”
Ticket 760
Lines beginning with ‘–’ are comments. To enable library profiling one has to uncomment the line
-- library-profiling: Falseand change the value of the option to True:
library-profiling: TrueThis is somewhat confusing: an option can be in three different states: True, False, and commented out.
Ticket 774
We can augment the tokenizer so that lines beginning with ‘>’ are left intact:
tokeniseLine :: (LineNo, Indent, HasTabs, String) -> [Token]
tokeniseLine (n0, i, t, l) = case split n0 l of
(Span _ l':ss) -> Line n0 i t l' :ss
cs -> cs
where split _ "" = []
split n s@('>' : _) = [Line n i t s]
split n s = case span (\c -> c /='}' && c /= '{') s of
("", '{' : s') -> OpenBracket n : split n s'
(w , '{' : s') -> mkspan n w (OpenBracket n : split n s')
("", '}' : s') -> CloseBracket n : split n s'
(w , '}' : s') -> mkspan n w (CloseBracket n : split n s')
(w , _) -> mkspan n w []
mkspan n s ss | null s' = ss
| otherwise = Span n s' : ss
where s' = trimTrailing (trimLeading s)Is it a good / right solution? Am I overlooking anything?
2013-05-04 Sat: Roman has pointed out that Duncan Coutts announced he was rewriting the parser of .cabal files, so it probably doesn’t makes sense to fix the old parser if it’s going to be replaced by the new one.
Ticket 757
Can’t reproduce with Firefox 20.0 (Linux/x86$_{\mathrm{64}}$).
Ticket 735
I tried to reproduce this issue and I can’t. I tried to build NaturalSort-0.2.1 with GHC 7.4.1 and Cabal 1.16.03, and I would expect the build to fail due to this line in NaturalSort.cabal: Cabal-Version: >= 1.6 && < 1.9, but it does build, and the log file does contain some log info. No summary file has been produced though.
I also tried to build cal3d, which on my machine fails due to a missing C library, but again the log file contains the error message. The summary file is generated when I try to install the library from Hackage but is not generated when I manually download the source and try to build the library locally, which confirms the behavior observed in Issue #1189.
Ticket 676
The culprit of the problem was the function simplifyCondTree, which traversed CondTree in post-order, putting global options after the options defined in the branches of conditionals. I’ve changed that to pre-order traversal.
Sent a pull request. Also, added a test demonstrating what’s changed.
Ticket 885
Duplicate of Issue 774. 2013-05-04 Sat: Closed by Johan Tibell.
Ticket 883
I don’t think this is a cabal bug.
I tried to reproduce this behaviour: I downloaded monad-par-0.3.4.1 and mangled the file Control/Monad/Par/Scheds/Direct.hs so that its top looked as follows:
{-# LANGUAGE RankNTypes, NamedFieldPuns, BangPatterns,
ExistentialQuantification, CPP, ScopedTypeVariables,
TypeSynonymInstances, MultiParamTypeClasses,
GeneralizedNewtypeDeriving, PackageImports,
ParallelListComp
#-} # OPTIONS_GHC
{- -Wall -fno-warn-name-shadowing -fno-warn-unused-do-bind #-}Running GHC directly on this file gives the following error message:
Control/Monad/Par/Scheds/Direct.hs:6:18: parse error on input `#'Building through cabal build produces the following error message:
Control/Monad/Par/Scheds/Direct.hs:1:1:
File name does not match module name:
Saw: `Main'
Expected: `Control.Monad.Par.Scheds.Direct'However, running GHC on Control/Monad/Par.hs produces an identical error message.
I’m guessing that something like this is happening:
- when GHC is run directly on
Control/Monad/Par/Scheds/Direct.hs, it tries to parse a module declaration, encounters# OPTIONS_GHC, decides that the file doesn’t have a module header (in particular, defaults the module name to Main), tries to parse# OPTIONS_GHCand fails; - when GHC is run on
Control/Monad/Par.hs, it expects to find a module namedControl.Monad.Par.Scheds.Directin the fileControl/Monad/Par/Scheds/Direct.hs, but when it tries to parse it, it encounters# OPTIONS_GHC, and decides that the file doesn’t have a module header, defaults the module name to Main, and fails with a different error.
This theory is supported by the following experiment: replace the pragmas and # OPTIONS_GHC above with some code, e.g., foo = 3. Then running GHC on Control/Monad/Par/Scheds/Direct.hs produces
Control/Monad/Par/Scheds/Direct.hs:12:1:
parse error on input `module'and running GHC on Control/Monad/Par.hs produces
Control/Monad/Par/Scheds/Direct.hs:1:1:
File name does not match module name:
Saw: `Main'
Expected: `Control.Monad.Par.Scheds.Direct'The summary of the above is that this is not a cabal bug (and probably not a bug at all), and this ticket should be closed.
2013-05-04 Sat: Closed by Johan Tibell.
Ticket 417
Is this still an issue? I see a comment in Cabal/Distribution/PackageDescription/Parse.hs:
-- "build-depends" is a local field now. To be backwards
-- compatible, we still allow it as a global field in old-style
-- package description files and translate it to a local field by
-- adding it to every non-empty sectionwhich makes me think that what Duncan is suggesting:
or it should automatically propagate such global options into each buildable component.
has already been implemented.
2013-05-05 Sun: It seems I was wrong, investigating.
Cabal Bug Fixing II
- Work on adding a –lowest-dependencies flag for cabal which builds a package against its lowest dependencies. This is useful for package maintainers to see whether the lower bounds of a package are still valid. An initial version is issued as a pull-request but the actual workings are still being tweaked.
- Small bugfix making the –package-db flag accept relative paths
Cabal Bug Fixing III
I took a look at Ticket 1297.
My first step was to grab the latest sources and run the tests.
bash-3.2$ cabal test
Running 2 test suites...
Test suite package-tests: RUNNING...
Cabal test suite - testing cabal version 1.17.0
BuildTestSuiteDetailedV09:
: [Failed]
build failed!
expected: False
but got: True
BuildDeps/InternalLibrary2:
: [Failed]
executable should have linked with the internal library
expected: "myLibFunc internal"
but got: ""
BuildDeps/InternalLibrary3:
: [Failed]
executable should have linked with the internal library
expected: "myLibFunc internal"
but got: ""
BuildDeps/InternalLibrary4:
: [Failed]
executable should have linked with the installed library
expected: "myLibFunc installed"
but got: ""
PackageTests/CMain:
: [OK]
Test Cases Total
Passed 20 20
Failed 4 4
Total 24 24
Test suite package-tests: FAIL
Test suite logged to: dist/test/Cabal-1.17.0-package-tests.log
Test suite unit-tests: RUNNING...
Test suite unit-tests: PASS
Test suite logged to: dist/test/Cabal-1.17.0-unit-tests.log
1 of 2 test suites (1 of 2 test cases) passed.BuildTestSuiteDetailedV09 is a test which tests that Cabal testing is working correctly. It takes a small Cabal project with a test-suite stanza builds it and runs the tests in the test suite.
This test passes on linux so it looked like the problem was in the way cabal was running the tests. This turned out to be a red herring. I pulled the test out of the Cabal test suite and ran it on its own (obvious in hindsight). So the problem was in Cabal not in the way Cabal was being tested.
bash-3.2$ ../cabal/cabal-install/dist/build/cabal/cabal configure --enable-tests --package-db=../cabal/Cabal/dist/package.conf.inplace
Warning: The package list for 'hackage.haskell.org' is 38 days old.
Run 'cabal update' to get the latest list of available packages.
Warning: my.cabal: A package using section syntax must specify at least
'cabal-version: >= 1.2'.
Resolving dependencies...
Warning: my.cabal: A package using section syntax must specify at least
'cabal-version: >= 1.2'.
Configuring BuildTestSuiteDetailedV09-0.1...
bash-3.2$ ../cabal/cabal-install/dist/build/cabal/cabal build
Warning: my.cabal: A package using section syntax must specify at least
'cabal-version: >= 1.2'.
Building BuildTestSuiteDetailedV09-0.1...
Preprocessing library BuildTestSuiteDetailedV09-0.1...
[1 of 1] Compiling Dummy ( Dummy.hs, dist/build/Dummy.o )
In-place registering BuildTestSuiteDetailedV09-0.1...
Preprocessing test suite 'dummy' for BuildTestSuiteDetailedV09-0.1...
[1 of 1] Compiling Dummy ( Dummy.hs, dist/build/Dummy.o )
ar: dist/build/Dummy.o: No such file or directoryBut why was Cabal complaining about a missing Dummy.o when it had clearly just created it?
Sasha suggested using strace (http://en.wikipedia.org/wiki/Strace) but this didn’t exist on my Mac. But I did have dtrace (http://hints.macworld.com/article.php?story=20071031121823710 and http://chihungchan.blogspot.com/2009/05/which-process-deleted-my-file.html).
Dominics-MacBook-Pro:Dummy dom$ dtrace -qn 'syscall::unlink:entry {printf("PID=%d, CMD=%s, FILE=%s\n", pid, curpsinfo->pr_psargs, copyinstr(arg0));}'
dtrace: failed to initialize dtrace: DTrace requires additional privileges
Dominics-MacBook-Pro:Dummy dom$ sudo dtrace -qn 'syscall::unlink:entry {printf("PID=%d, CMD=%s, FILE=%s\n", pid, curpsinfo->pr_psargs, copyinstr(arg0));}'
Password:
PID=16137, CMD=Emacs, FILE=/Users/dom/Dropbox/Private/.#Everything.org
PID=16137, CMD=Emacs, FILE=/Users/dom/Dropbox/Private/#Everything.org#
PID=82566, CMD=ar, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T//ar.ugfvvt
PID=82567, CMD=libtool, FILE=dist/build/libHSBuildTestSuiteDetailedV09-0.1.a
PID=82557, CMD=cabal, FILE=dist/build/libHSBuildTestSuiteDetailedV09-0.1.a
PID=82557, CMD=cabal, FILE=dist/build/libHSBuildTestSuiteDetailedV09-0.1_p.a
PID=82557, CMD=cabal, FILE=dist/build/libHSBuildTestSuiteDetailedV09-0.1-ghc7.6.2.dylib
PID=82557, CMD=cabal, FILE=dist/build/HSBuildTestSuiteDetailedV09-0.1.o
PID=82560, CMD=ghc, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T/ghc82560_0/ghc82560_0.c
PID=82560, CMD=ghc, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T/ghc82560_0/ghc82560_0.s
PID=82573, CMD=i686-apple-darwi, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T//ccnsroxB.s
PID=82562, CMD=i686-apple-darwi, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T//ccjXqsIc.s
PID=82571, CMD=ghc, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T/ghc82571_0/ghc82571_0.c
PID=82571, CMD=ghc, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T/ghc82571_0/ghc82571_0.s
PID=82576, CMD=as, FILE=dist/build/Dummy.o
PID=82577, CMD=ar, FILE=/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T//ar.mlwmuT
PID=82557, CMD=cabal, FILE=dist/build/libdummy.a
PID=82557, CMD=cabal, FILE=dist/build/libdummy_p.a
PID=82557, CMD=cabal, FILE=dist/build/libdummy-ghc7.6.2.dylib
PID=82557, CMD=cabal, FILE=dist/build/dummy.oI noticed I had a Dummy.o and a dummy.o. Also the name of the test-suite was dummy and the name of the module in the the Cabal package was Dummy. I tried changing the name of the test-suite stanza to Dummy. No luck but when I tried changing the name of the test-suite stanza to something completely different everything worked.
bash-3.2$ ../cabal/cabal-install/dist/build/cabal/cabal build
Warning: my.cabal: A package using section syntax must specify at least
'cabal-version: >= 1.2'.
Building BuildTestSuiteDetailedV09-0.1...
Preprocessing library BuildTestSuiteDetailedV09-0.1...
[1 of 1] Compiling Dummy ( Dummy.hs, dist/build/Dummy.o )
In-place registering BuildTestSuiteDetailedV09-0.1...
Preprocessing test suite 'Urk' for BuildTestSuiteDetailedV09-0.1...
[1 of 1] Compiling Dummy ( Dummy.hs, dist/build/Dummy.o )
In-place registering Urk-0.1...
[1 of 1] Compiling Main ( dist/build/UrkStub/UrkStub-tmp/UrkStub.hs, dist/build/UrkStub/UrkStub-tmp/Main.o )
Linking dist/build/UrkStub/UrkStub ...Case Insensitive but Case Preserving
It turns out that most Macs are configured like this. You can check as shown below. The absence of Case-sensitive in the Name field means the file system is case insensitive.
bash-3.2$ diskutil info disk0s2
Device Identifier: disk0s2
Device Node: /dev/disk0s2
Part of Whole: disk0
Device / Media Name: Customer
Volume Name: Macintosh HD
Escaped with Unicode: Macintosh%FF%FE%20%00HD
Mounted: Yes
Mount Point: /
Escaped with Unicode: /
File System Personality: Journaled HFS+
Type (Bundle): hfs
Name (User Visible): Mac OS Extended (Journaled)
Journal: Journal size 40960 KB at offset 0x1238b000
Owners: Enabled
Partition Type: Apple_HFS
OS Can Be Installed: Yes
Media Type: Generic
Protocol: SATA
SMART Status: Verified
Volume UUID: 86F67826-3F78-387D-B335-C695CADCFD86
Total Size: 499.4 GB (499418034176 Bytes) (exactly 975425848 512-Byte-Blocks)
Volume Free Space: 427.8 GB (427780079616 Bytes) (exactly 835507968 512-Byte-Blocks)
Device Block Size: 512 Bytes
Read-Only Media: No
Read-Only Volume: No
Ejectable: No
Whole: No
Internal: Yes
Solid State: YesCoda
Changing dummy to Dummy tickles the bug on linux so now we have an easily reproducible bug although it is not clear how to fix this: Ticket 1311.
and 
are column vectors of size 
is a function such as the logistic function 
or 
.
observations the likelihood is:
is a guess for the parameters of the model then we can improve the guess by stepping a small distance in the direction of greatest change.
is some constant known in machine learning as the learning rate. It must be chosen to be large enough to ensure convergence within a reasonable number of steps but not so large that the algorithm fails to converge.
has the label 1 given the observations 
.
.
runs from 1 to 
are i.i.d. normal with mean 
and the same variance 
: 
;
is to make the manipulation of the derivative easier. In our case, this is irrelevant as we are not going to derive the derivative explicitly but use automated differentiation.
given the observations 
is
at time 
.
is the 
let:
. Away from the sampling times there is no dependence on 
so we diffuse one pricer back to the start time which gives us the price of the option for any given 
and 
as we will want to use our pricer not just for a vanilla call option.
, we have a simpler situation.
so we can take the diagonal of our grid and diffuse backwards using a single pricer.
and put these into a histogram.
