what is the pipestat store attribute?

[nsheff]

the pipestat.PipestatManager has a store attribute:

https://github.com/pepkit/pipestat/blob/dev/pipestat/pipestat.py#L90

It appears to be being used to set/get items from a dict:

pipestat/pipestat/pipestat.py

Lines 234 to 247 in f228c0b

	def __getitem__(self, key):
	return self.store[self._keytransform(key)]
	def __setitem__(self, key, value):
	self.store[self._keytransform(key)] = value
	def __delitem__(self, key):
	del self.store[self._keytransform(key)]
	def __iter__(self):
	return iter(self.store)
	def __len__(self):
	return len(self.store)

I can't see where this is connected to the backends. What is this store for? Is this a relic that should be removed?

[donaldcampbelljr]

This was actually added when we decided to change PipestatManger to inherit from MutableMapping and not dict. Essentially, the above code has pipestat act like a dict, but it is not technically a dict.

[nsheff]

Ok, interesting.

What purpose does it serve? It is not connected to the back-end, so this seems just... superfluous.

[donaldcampbelljr]

It is so that PipestatManager can store attributes like so:

pipestat/pipestat/pipestat.py

Lines 151 to 162 in f228c0b

	self[STATUS_FILE_DIR] = mk_abs_via_cfg(flag_file_dir, self.config_path or self.file)
	self.backend = FileBackend(
	self[FILE_KEY],
	record_identifier,
	self[PIPELINE_NAME],
	self[PIPELINE_TYPE],
	self[SCHEMA_KEY],
	self[STATUS_SCHEMA_KEY],
	self[STATUS_FILE_DIR],
	self[RESULT_FORMATTER],
	self[MULTI_PIPELINE],
	)

pipestat/pipestat/pipestat.py

Lines 662 to 668 in f228c0b

	def record_identifier(self) -> str:
	"""
	Pipeline type: "sample" or "project"
	:return str: pipeline type
	"""
	return self.get(RECORD_IDENTIFIER)

[nsheff]

I see.

I propose we change this so that:

1. For internal config metadata, we use a dedicated dict and don't use getitem syntax, so the above would be changed to: self.config[FILE_KEY] and self.config.get(RECORD_IDENTIFIER). Since this seems to be for internal use, I'm not sure we need the getitem convenience.
2. We change the getitem syntax to actually be an alias for .retrieve, and setitem to be an alias for report. This would be a simplified form, less powerful than using the actual retrieve, but maybe more convenient.

To insert a result then, you'd use psm["my_result_key"] = my result_value.

Why? Because then a PipestateManager object is actually a Mapping of the values it is reporting to the database. This would allow me to use it in seqcol instead of my RDBDict object.

Alternatively, I could just write a little wrapper around PipestatManager that would do this.

[nsheff]

any objection to this @donaldcampbelljr ?

[nsheff]

Related to #97

[donaldcampbelljr]

No objection. I think this is a cool idea. I can begin implementing it this week.

[donaldcampbelljr]

Ok, I have added the discussed changes for this. I opted to use psm.cfg instead of psm.config.

Changes exist in branch dev_99_103_API_Refactor

[nsheff]

excellent, thanks

[donaldcampbelljr]

Closed via #92