The sdist command in distutils calculates the MANIFEST file from a template (by default MANIFEST.in). 

The code in sdist assumes that the contents of MANIFEST only depends on MANIFEST.in and setup.py, which can cause files to be missed when building an sdist.

In particular: given a MANIFEST.in file that includes all documentation in a 'Doc' subdirectory, using"recursive-include Doc *.txt" in the template, the MANIFEST won't get regenerated when a new file is added to the documentation subdirectory and hence that new file is not included in sdist distributions.

One way to fix this: is to always recreate MANIFEST when an explicit MANIFEST.in file exists, as in the attached patch.

(The patch is not perfect: I'd rename "template_newer" before committing)

The same problem occurs without a manifest template: you can have options or imports in setup.py that will change the MANIFEST file. 

For example : if setup() has:

  packages=['foo']

And if you add a subpackage "bar" in "foo" with some modules, with an existing MANIFEST, they won't be added.

IOW, the MANIFEST file should be recalculated *everytime*. I am going to remove all the "don't rebuilt MANIFEST" code in Distutils *and* Distutils2.

done in r81255, r81256 (2.6), r81258 (py3), r81260 (3.1)

Thanks Ronald

This change just wrecked Mercurial's release build process.

We've been building Mercurial release tarballs with a Makefile target wrapped around sdist for most of five years, and we've never had or wanted a MANIFEST.in file. We generate an exact, complete, and correct MANIFEST ourselves with:

 hg manifest > MANIFEST

This fix ignores that MANIFEST entirely and attempts to build a new one even though MANIFEST.in doesn't exist.

So not only does this change years of documented behavior in a way that very nearly caused us to ship a tarball missing over a thousand files, it means doing what we want to do with sdist (ship precisely the files listed in our version control manifest) is now significantly harder.

Unfortunately, this fix is now in a long-lived Python release which we support, which means we're stuck with it.

this fix was done to avoid generating broken MANIFEST file with sdist.

IIUC your problem is more about avoiding generating a MANIFEST file at *all* via sdist, so we should add a --no-manifest option to the sdist command.

This can be added today in Mercurial setup.py file as well so you don't have the problem with 2.7. I can contribute this change if you want.

Ok, we need a change that will work with Python 2.4 through 2.7.

Follow-up at http://selenic.com/pipermail/mercurial-packaging/2010-August/000009.html

This issue has broken our build system too.

A bug has been fixed, but another one is open now.

If the MANIFEST file has been produced by distutils it should be
regenerated each time.  But if it has not been produced by distutils
then it should not regenerated.

The problem is how to make the distinction.  Maybe an special line at
the beginning of the MANIFEST file could identify that it has been
generated by distutils.

I agree with msg113530 and msg112371 that the current behavior is a bug: distutils is now too agressive because it recalculates the MANIFEST file even when distutils isn't the one that calculated it in the first place.

This is a functional regression w.r.t. earlier python releases.

I like the idea of a marker in a comment line in the MANIFEST, we can introduce this as a bug fix to avoid this regression

Attaching a patch to implement the marker idea. No doc changes yet.

It should work transparently: you don’t have to give an option to have automatic recalculation, you don’t have to give an option to have it leave your manual MANIFEST alone.

(For the curious, history is at http://beta.intuxication.org/hg/merwok/python-patches. Yay for TDD and Mercurial.)

New version of the patch to address Tarek’s comments and add NEWS and docs entries.

This is a regression introduced in 2.6.6rc1.  After discussion in irc with merwok, it is agreed that he will revert to pre-2.6.6rc1 behavior and not apply any fix.  It's way too late to be changing behavior in 2.6.6.  Folks using Python 2.6 will just have to live with the devil they know.

2.6 regression reverted in r83992.

Tarek is okay with my patch for the other branches. I noticed the first fix did not have a versionadded in the docs, so I’ll add that and commit.

Patch committed in r83993 (py3k), r83994 (3.1) and r83996 (2.7).

I got the version wrong in 3.x and fixed it in r83995 and r83998.

Patch ported to distutils2 (http://bitbucket.org/tarek/distutils2/changeset/5abba4a77f5a). Closing.

Please reopen with priority set to “release blocker” if 2.6 behaves badly before Monday. Thanks.

There is a bug in the new code.  See #11104.