Opened 12 years ago

Closed 11 years ago

#380 closed task (fixed)

copy cache

Reported by: Peter Johansson Owned by: Peter Johansson
Priority: major Milestone: svndigest 0.8
Component: core Version: trunk
Keywords: Cc:

Description

The use case is when, e.g., creating a branch and suddenly having two wc.

Change History (27)

comment:1 Changed 12 years ago by Peter Johansson

Milestone: svndigest 0.x+svndigest 0.8
Owner: changed from Jari Häkkinen to Peter Johansson

comment:2 Changed 11 years ago by Peter Johansson

Milestone: svndigest 0.8svndigest 0.9

comment:3 Changed 11 years ago by Peter Johansson

Milestone: svndigest 0.9svndigest 0.8

How we currently copy the cache in ReleaseProcedure (copying the entire wc) is ridiculous so I move this ticket to version 0.8. It should be an easy task to write a shell script doing the copying.

comment:4 in reply to:  3 ; Changed 11 years ago by Jari Häkkinen

Replying to peter:

How we currently copy the cache in ReleaseProcedure (copying the entire wc) is ridiculous so I move this ticket to version 0.8. It should be an easy task to write a shell script doing the copying.

There is no cache information in ReleaseProcedure for svndigest but for yat, http://dev.thep.lu.se/yat/wiki/ReleaseProcedure. And you are right, we should have a script for that. For beauty we will pay.

comment:5 in reply to:  4 Changed 11 years ago by Peter Johansson

Replying to jari:

Replying to peter:

How we currently copy the cache in ReleaseProcedure (copying the entire wc) is ridiculous so I move this ticket to version 0.8. It should be an easy task to write a shell script doing the copying.

There is no cache information in ReleaseProcedure for svndigest but for yat,

Hmm, in my mind I have already updated RP... (#257 #399)

comment:6 Changed 11 years ago by Peter Johansson

Status: newassigned

I'm thinking of an interface that looks something like this:

Known options are:
  -h, --help            print this help and exit
  -r, --root=ROOT       directory to copy cache from
  -t, --target=TARGET   directory to copy cache to
      --version         print version and exit

We find all files under ROOT with name '*.svndigest-cache' residing in a directory '.svndigest', and copy those files to the corresponding location in TARGET.

Would this be OK?

comment:7 Changed 11 years ago by Jari Häkkinen

There are many errors that may happen. What if the directory structure is not the same? How does the cache usage in svndigest work if cache information is not syncronized, i.e., due to some freak event cache level may be different between files and directories?

comment:8 in reply to:  7 ; Changed 11 years ago by Peter Johansson

Replying to jari:

There are many errors that may happen.

Indeed

What if the directory structure is not the same?

I think it makes sense to create the mother directory '.svndigest', but if its mother directory does not exists, I think we exit 1.

How does the cache usage in svndigest work if cache information is not syncronized, i.e., due to some freak event cache level may be different between files and directories?

Only files are cached. Directories get their stats from daughter nodes, and this is true both when using cache and when talking with svn repo.

comment:9 in reply to:  8 ; Changed 11 years ago by Jari Häkkinen

Replying to peter:

How does the cache usage in svndigest work if cache information is not syncronized, i.e., due to some freak event cache level may be different between files and directories?

Only files are cached. Directories get their stats from daughter nodes, and this is true both when using cache and when talking with svn repo.

Ok, but can files in the same directory have different cache level? Or is the same cached revision expected?

comment:10 in reply to:  9 ; Changed 11 years ago by Peter Johansson

Replying to jari:

Ok, but can files in the same directory have different cache level? Or is the same cached revision expected?

The cache of each file reflects the revision of that particular file. So I guess you can have a cache structure reflecting a mixture of revisions. This could probably be problematic, although I cannot see the exactly what the problem would be, but this is not a problem only here. I've experienced problem when running copyright on a mixed revision and often resulting in an assert. It might be a good idea to check for mixed revisions already in Parameter::analyse, or is it a desired feature?

comment:11 in reply to:  10 ; Changed 11 years ago by Jari Häkkinen

Replying to peter:

Replying to jari: The cache of each file reflects the revision of that particular file. So I guess you can have a cache structure reflecting a mixture of revisions. This could probably be problematic, although I cannot see the exactly what the problem would be, but this is not a problem only here. I've experienced problem when running copyright on a mixed revision and often resulting in an assert. It might be a good idea to check for mixed revisions already in Parameter::analyse, or is it a desired feature?

If we cannot guarantee proper functionality with mixed revision cache then we should check for that.

My worry is that if we start to copy cache then one could imagine that one sub-tree has a copied cache whereas another has none (or another cache). But, maybe I misunderstand what copy cache is supposed to accomplish? What are the restrictions on the source and target? Should we use source instead of root?

comment:12 in reply to:  11 ; Changed 11 years ago by Peter Johansson

Replying to jari:

Replying to peter:

Replying to jari: The cache of each file reflects the revision of that particular file. So I guess you can have a cache structure reflecting a mixture of revisions. This could probably be problematic, although I cannot see the exactly what the problem would be, but this is not a problem only here. I've experienced problem when running copyright on a mixed revision and often resulting in an assert. It might be a good idea to check for mixed revisions already in Parameter::analyse, or is it a desired feature?

If we cannot guarantee proper functionality with mixed revision cache then we should check for that.

I opened a ticket:415 for that.

My worry is that if we start to copy cache then one could imagine that one sub-tree has a copied cache whereas another has none (or another cache).

I don't see a problem with that. The cache file is used by the Stats class that uses it as follows:

  • If no cache file get stats from repo and we are done
  • else build stats from cache file
  • if cache is uptodate, i.e., it reflects same revision as file then we are done.
  • else talk to repo to get stats for remaining revisions

This happens for each file independently, so I don't see a problem with having some files cached and some not. The files that have cache will use its cache and those which don't won't. Or am I'm missing something here?

But, maybe I misunderstand what copy cache is supposed to accomplish?

I don't know; that is as always hard to tell ;)

What are the restrictions on the source and target?

Well in the implementation I'm working on I'm using find and cp, so the obvious requirements are that source is readable and target is read/writable. I have no plans in making the copy atomic, but files will be copied sequentially and when an error occurs the script exits.

Other restriction is that directories must pre-exist in target, with the exception of .svndigest directories. These will be created because the typical use case is that target is a fresh pristine wc.

Should we use source instead of root?

Sure. I suggested --root because that is what we have in svndigest, but we can use --source. Is that inspired from cp? Should we use destination as well?

comment:13 Changed 11 years ago by Peter Johansson

(In [907]) Autoconf 2.63 is now required (closes #414).

refs #380. A first implementation of svndigest-copy-cache. It is is written in M4sh. Autoconf compiles the source in svndigest-copy-cache.as to svndigest-copy-cache.in, which in turn is processed by config.status to replace some keywords such as PACKAGE and friends. There is no docs and no test yet.

added *.as in config file

comment:14 in reply to:  12 ; Changed 11 years ago by Jari Häkkinen

Replying to peter:

Sure. I suggested --root because that is what we have in svndigest, but we can use --source. Is that inspired from cp? Should we use destination as well?

If we claim cp inspiration we should stick with target. One thought, will it be confusing to have root/target for svndigest and source/destination for copy cache? Hm, maybe we should stick with root/target?

comment:15 Changed 11 years ago by Jari Häkkinen

bootstrap is complaining

svndigest> ./bootstrap 
autom4te: cannot open svndigest-copy-cache.in: Permission denied
autom4te: cannot open svndigest-copy-cache.in: Permission denied
autom4te: cannot open svndigest-copy-cache.in: Permission denied
svndigest> autoreconf --version
autoreconf (GNU Autoconf) 2.64
Copyright (C) 2009 Free Software Foundation, Inc.

Does this issue belong to this ticket?

comment:16 in reply to:  15 Changed 11 years ago by Peter Johansson

Replying to jari:

bootstrap is complaining

svndigest> ./bootstrap 
autom4te: cannot open svndigest-copy-cache.in: Permission denied
autom4te: cannot open svndigest-copy-cache.in: Permission denied
autom4te: cannot open svndigest-copy-cache.in: Permission denied
svndigest> autoreconf --version
autoreconf (GNU Autoconf) 2.64
Copyright (C) 2009 Free Software Foundation, Inc.

Does this issue belong to this ticket?

Yes, most likely. What are the permissions on svndigest-copy-cache.in when you get that error? Does it exist?

comment:17 Changed 11 years ago by Peter Johansson

(In [911]) refs #380. Changing permission mode to 'rw' for user to avoid problem. Hope this fixes the problem with 'cannot open svndigest-copy-cache.in'. You need to chmod or remove the file in question before bootstrapping.

comment:18 Changed 11 years ago by Jari Häkkinen

svndigest-copy-cache.in existed and was read only. Removed the file updated and all is well now.

comment:19 Changed 11 years ago by Peter Johansson

(In [912]) refs #380. Testing that svndigest-copy-cache runs and supports --version and --help with no output to stderr

comment:20 Changed 11 years ago by Peter Johansson

(In [913]) refs #380. Use the RELEASE_YEAR variable in svndigest-copy-cache

comment:21 Changed 11 years ago by Peter Johansson

(In [914]) refs #380. removed invalid comment about absent grandma and added explaining comment.

comment:22 in reply to:  14 ; Changed 11 years ago by Peter Johansson

Replying to jari:

Replying to peter:

Sure. I suggested --root because that is what we have in svndigest, but we can use --source. Is that inspired from cp? Should we use destination as well?

If we claim cp inspiration we should stick with target. One thought, will it be confusing to have root/target for svndigest and source/destination for copy cache? Hm, maybe we should stick with root/target?

I have no strong opinion, but I would vote for root/target. We should nail a decision here before implementing a possible change.

comment:23 in reply to:  22 Changed 11 years ago by Jari Häkkinen

Replying to peter:

I have no strong opinion, but I would vote for root/target. We should nail a decision here before implementing a possible change.

Go for root/target.

comment:24 Changed 11 years ago by Peter Johansson

(In [923]) refs #380. Change back to --root (and -r).

comment:25 Changed 11 years ago by Peter Johansson

(In [957]) refs #380. extended test and fixed detected bug in cmd opt parsing

comment:26 Changed 11 years ago by Peter Johansson

(In [958]) refs #380. cleaning up after finished test

comment:27 Changed 11 years ago by Peter Johansson

Resolution: fixed
Status: assignedclosed

(In [959]) documenting svndigest-copy-cache. closes #380

Note: See TracTickets for help on using tickets.